1 Domain problem formulation

What is the real-world question? This could be hypothesis-driven or discovery-based.

Some advice

Why is this question interesting and important? What are the implications of better understanding this data?

Briefly describe any background information necessary to understand this problem.

Some advice

Briefly describe how this question can be answered in the context of a model or analysis.

Outline the rest of the report/analysis.

2 Data

What is the data under investigation? Provide a brief overview/description of the data.

Describe how your data connects to the domain problem.

2.1 Data Collection

How was the data collected or generated (including details on the experimental design)? Be as transparent as possible so that conclusions made from this data are not misinterpreted down the road.

Describe any limitations when using the data to answer the domain problem of interest.

Where is the data stored, and how can it be accessed by others (if applicable)?

2.2 Data Splitting

TODO: add advice for possible data splits, AK getting nice figure together

Decide on the proportion of data in each split.

Decide on the “how” to split the data (e.g., random sampling, stratified sampling, etc.), and explain why this is a reasonable way to split the data.

Split the data into a training, validation, and test set.

data_split <- dataSplit(X = X, y = y, stratified_by = y,
                        train_prop = params$train_prop, 
                        valid_prop = params$valid_prop, 
                        test_prop = params$test_prop)
Xtrain <- data_split$X$train
Xvalid <- data_split$X$validate
Xtest <- data_split$X$test
ytrain <- data_split$y$train
yvalid <- data_split$y$validate
ytest <- data_split$y$test

Provide summary statistics and/or figures of the three data sets to illustrate how similar (or different) they are.

2.2.1 Data Splitting Overview

X Data Split

plotDataSplit(Xtrain, Xvalid, Xtest, 
              xlab = "X", title = "Overall X Distribution")

Y Data Split

plotDataSplit(ytrain, yvalid, ytest, 
              xlab = "y", title = "Overall y Distribution")

2.3 Data Cleaning and Preprocessing

What steps were taken to clean the data? More importantly, why was the data cleaned in this way?

Discuss all inconsistencies, problems, oddities in the data (e.g., missing data, errors in data, outliers, etc.).

Record your preprocessing steps in a way such that if someone else were to reproduce your analysis, they could easily replicate and understand your steps.

It can be helpful to include relevant plots that explain/justify the choices that were made when cleaning the data.

If more than one preprocessing pipeline is reasonable, examine the impacts of these alternative preprocessing pipelines on the final data results.

Again, be as transparent as possible. This allows others to make their own educated decisions on how best to preprocess the data.

## DO DATA CLEANING / PRE-PROCESSING HERE 
Xtrain <- log(Xtrain + 1) %>%
  removeConstantCols(verbose = 1) %>%
  removeDuplicateCols(verbose = 1) %>%
  filterColsByVar(max_p = 1000)
#> Removed 15 features with constant values
#> Removed 5 features that are duplicated
Xvalid <- log(Xvalid + 1)[, colnames(Xtrain)]
Xtest <- log(Xtest + 1)[, colnames(Xtrain)]

2.4 Data Exploration

TODO: Add drag and drop feature in shiny version for other images

The main goal of this section is to give the reader a feel for what the data “looks like” at a basic level.

Provide plots that summarize the data and perhaps even plots that convey some smaller findings which ultimately motivate the main findings.

Provide additional plots representing remaining oddities after pre-processing if applicable.

Add summary statistics in accompanying tables (or in figures) for quick comparisons.

2.4.1 Data Overview

#> Number of features: 1000
#> Number of training samples: 649
#> Number of validaiton samples: 216
#> Number of test samples: 218

# summary of types of features in (X, y) data
dataTypes(X = Xtrain, y = ytrain)

Summary Tables

# broad array of summary statistics for features in training (X, y)
tab_ls <- dataSummary(X = Xtrain, y = ytrain)
for (dtype in names(tab_ls)) {  # grouped by data type
  simChef:::subchunkify(tab_ls[[dtype]], i = chunk_idx, 
                        other_args = "results='asis'")
  chunk_idx <- chunk_idx + 1
}

X Distribution

# plot X distribution
plotDataDistribution(data = Xtrain, 
                     xlab = "X", title = "Training X Distribution")

Y Distribution

# plot y distribution
plotDataDistribution(data = ytrain, 
                     xlab = "y", title = "Training y Distribution")

Data Heatmap

# (clustered) heatmap of (X, y) data
plotDataHeatmap(X = Xtrain, y = ytrain, clust_rows = TRUE, clust_cols = TRUE,
                show_ytext = FALSE, x_text_angle = TRUE)

Feature Correlation

# (clustered) correlation heatmap
plotCorHeatmap(X = Xtrain, cor_type = "pearson", clust = TRUE,
               x_text_angle = TRUE) +
  ggplot2::labs(x = "Features", y = "Features", fill = "Cor.")

Feature Pair Plots

# feature pair plot (for a subset of features)
keep_features <- sort(
  sample(1:ncol(Xtrain), min(ncol(Xtrain), 6), replace = FALSE)
)
plotPairs(data = Xtrain, columns = keep_features, 
          color = ytrain, color_label = "y")

Marginal Associations

# marginal association plots (for a subset of features)
caret::featurePlot(x = Xtrain[, keep_features],
                   y = ytrain,
                   plot = if (is.factor(ytrain)) "box" else "scatter",
                   scales = list(x = list(relation = "free"), 
                                 y = list(relation = "free")))

PCA

# pca plot
plotPCA(X = Xtrain, npcs = 3, color = ytrain, color_label = "y",
        center = TRUE, scale = FALSE)$plot

3 Prediction Modeling

TODO: add advice on which models to select and why

Discuss the prediction methods under consideration, and explain why these methods were chosen.

Discuss the accuracy metrics under consideration, and explain why these metrics were chosen.

Note: there should be multiple methods and metrics under consideration to paint a more holistic picture of the data. At least one method should be a baseline, common approach that may not be optimal for the problem setting, but serves as a helpful comparison.

3.1 Prediction check

Carry out the prediction pipeline, outlined above.

Fit prediction methods on training data.
Evaluate prediction methods on validation data.
Compare results, and filter out poor models.

## IF USING CARET MODELING BACKEND, CHOOSE METHODS AND TRAINING CONTROLS HERE
tr_control <- caret::trainControl(
  method = "cv",
  number = 5,
  classProbs = if (is.factor(ytrain)) TRUE else FALSE,
  summaryFunction = caret::defaultSummary,
  allowParallel = FALSE,
  verboseIter = FALSE
)

model_list <- list(
  ranger = list(tuneGrid = expand.grid(mtry = seq(sqrt(ncol(Xtrain)), 
                                                  ncol(Xtrain) / 3,
                                                  length.out = 3),
                                       splitrule = "gini",
                                       min.node.size = 1),
                importance = "impurity",
                num.threads = 1),
  xgbTree = list(tuneGrid = expand.grid(nrounds = c(10, 25, 50, 100, 150),
                                        max_depth = c(3, 6),
                                        colsample_bytree = 0.33,
                                        eta = c(0.1, 0.3),
                                        gamma = 0,
                                        min_child_weight = 1,
                                        subsample = 0.6),
                 nthread = 1)
)

start_time <- Sys.time()
model_results <- fitCaret(Xtrain = Xtrain, ytrain = ytrain, 
                          Xtest = Xvalid, ytest = yvalid, 
                          model_list = model_list, tr_control = tr_control)
#> [17:50:33] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:33] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:33] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:33] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:33] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:33] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:33] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:33] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:40] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:40] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:40] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:40] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:40] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:40] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:40] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:40] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:51] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:51] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:51] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:51] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:51] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:51] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:51] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:51] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:58] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:58] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:58] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:58] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:58] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:58] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:58] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:50:58] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:06] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:06] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:06] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:06] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:06] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:06] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:06] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:06] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:11] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:11] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:11] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:11] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:11] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:11] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:11] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:11] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:16] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:16] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:16] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:16] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:16] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:16] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:16] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:16] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:22] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:22] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:22] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:22] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:22] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:22] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:22] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:22] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:30] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:30] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:30] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:30] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:30] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:30] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:30] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:30] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:34] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:34] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:34] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:34] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:34] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:34] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:34] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:35] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:39] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:39] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:39] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:39] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:39] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:39] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:39] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:39] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:54] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:54] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:54] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:54] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:54] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:54] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:54] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:54] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:59] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:59] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:59] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:59] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:59] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:59] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:59] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:51:59] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:03] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:03] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:03] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:03] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:03] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:03] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:03] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:03] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:10] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:10] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:10] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:10] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:10] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:10] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:10] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:10] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:18] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:18] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:18] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:18] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:18] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:18] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:18] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:18] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:23] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:23] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:23] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:23] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:23] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:23] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:23] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:23] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:28] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:28] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:28] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:28] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:28] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:28] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:28] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:52:28] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
cat(sprintf("Fitting time taken: %f minutes", 
            difftime(Sys.time(), start_time, units = "mins")))
#> Fitting time taken: 4.946309 minutes

# table of accuracy metrics
prettyTable(model_results$errors,
            digits = 2, sigfig = FALSE, rownames = FALSE,
            caption = "Validation Prediction Accuracies", 
            html_options = list(options = list(dom = "t")))

3.2 Stability check

Taking the prediction methods that pass the prediction check, perform stability analysis.

Specify and justify the appropriate data perturbation(s).
Re-fit the prediction methods on these perturbed data sets.
Evaluate prediction methods on validation data.
Assess stability across the data perturbations as well as across the various methods.
Filter out poor models where necessary and interpret stability results.

TODO: Add results for h20, tuning needs to be fixed for h20 + tidymodels

n_reps <- 2 # increase for better stability measures when not testing code

# p <- progressr::progressor(steps = n_reps)
# future::plan(multisession, workers = min(n_reps, parallel::detectCores() - 1))
bootstrap_model_results <- future.apply::future_replicate(
  n = n_reps,
  expr = {
    bootstrap <- sample(1:nrow(Xtrain), nrow(Xtrain), replace = TRUE)
    Xtrain_b <- Xtrain[bootstrap, ]
    ytrain_b <- ytrain[bootstrap]
    
    if (params$modeling_pkg == "caret") {
      model_results_b <- fitCaret(Xtrain = Xtrain_b, ytrain = ytrain_b, 
                                  Xtest = Xvalid, ytest = yvalid, 
                                  model_list = model_list, 
                                  tr_control = tr_control)
    } else if (params$modeling_pkg == "tidymodels") {
      model_results_b <- fitTidyModels(Xtrain = Xtrain_b, ytrain = ytrain_b, 
                                       Xtest = Xvalid, ytest = yvalid,
                                       model_list = model_list, kfolds = kfolds)
    } else if (params$modeling_pkg == "h2o") {
      model_results_b <- fith2o(Xtrain = Xtrain_b, ytrain = ytrain_b,
                                Xtest = Xvalid, ytest = yvalid,
                                model_list = model_list)
    }
    return(model_results_b)
  },
  simplify = FALSE
)
#> [17:54:49] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:54:49] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:54:49] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:54:49] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:54:49] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:54:49] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:54:49] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:54:49] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:54:56] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:54:56] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:54:56] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:54:56] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:54:56] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:54:56] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:54:56] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:54:56] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:00] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:00] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:00] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:00] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:00] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:00] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:00] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:00] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:10] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:10] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:10] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:10] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:10] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:10] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:10] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:10] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:25] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:25] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:25] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:25] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:25] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:25] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:25] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:25] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:31] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:31] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:31] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:31] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:31] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:31] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:31] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:31] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:38] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:38] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:38] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:38] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:38] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:38] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:38] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:38] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:47] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:47] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:47] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:47] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:47] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:47] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:47] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:47] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:55:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:00] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:00] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:00] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:00] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:00] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:00] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:00] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:00] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:08] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:08] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:08] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:08] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:08] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:08] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:08] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:08] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:14] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:14] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:14] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:14] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:14] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:14] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:14] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:14] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:25] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:25] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:25] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:25] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:25] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:25] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:25] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:25] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:29] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:29] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:29] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:29] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:29] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:29] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:29] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:56:29] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:58:39] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:58:39] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:58:39] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:58:39] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:58:39] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:58:39] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:58:39] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:58:39] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:58:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:58:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:58:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:58:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:58:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:58:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:58:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:58:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:58:50] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:58:50] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:58:50] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:58:50] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:58:50] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:58:50] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:58:50] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:58:50] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:58:54] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:58:54] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:58:54] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:58:54] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:58:54] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:58:54] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:58:54] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:58:54] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:00] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:00] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:00] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:00] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:00] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:00] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:00] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:00] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:07] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:07] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:07] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:07] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:07] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:07] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:07] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:07] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:11] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:11] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:11] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:11] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:11] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:11] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:11] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:11] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:15] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:15] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:15] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:15] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:15] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:15] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:15] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:15] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:21] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:28] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:28] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:28] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:28] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:28] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:28] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:28] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:28] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:32] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:32] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:32] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:32] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:32] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:32] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:32] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:32] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:36] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:36] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:36] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:36] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:36] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:36] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:36] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:36] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:49] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:49] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:49] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:49] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:49] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:49] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:49] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:49] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:54] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:54] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:54] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:54] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:54] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:54] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:54] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:54] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:58] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:58] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:58] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:58] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:58] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:58] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:58] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [17:59:58] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:00:05] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:00:05] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:00:05] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:00:05] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:00:05] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:00:05] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:00:05] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:00:05] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:00:12] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:00:12] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:00:12] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:00:12] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:00:12] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:00:12] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:00:12] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:00:12] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:00:16] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:00:16] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:00:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:00:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:00:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:00:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:00:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:00:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:00:22] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:00:22] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:00:22] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:00:22] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:00:22] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:00:22] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:00:22] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:00:22] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.

bootstrap_model_errs <- purrr::map_dfr(bootstrap_model_results, "errors", 
                                       .id = "bootstrap_id")
bootstrap_model_preds <- purrr::map(bootstrap_model_results, "predictions",
                                    .id = "bootstrap_id")
bootstrap_model_imps <- purrr::map(bootstrap_model_results, "importance",
                                   .id = "bootstrap_id") %>%
  dplyr::bind_rows(.id = "bootstrap_id")

bootstrap_model_errs_summary <- bootstrap_model_errs %>%
  dplyr::group_by(model) %>%
  dplyr::summarise(dplyr::across(-bootstrap_id, list(mean = mean, sd = sd))) %>%
  setNames(stringr::str_replace(colnames(.), "_", " "))

# table of accuracy metrics
prettyTable(
  bootstrap_model_errs_summary,
  digits = 2, sigfig = FALSE, rownames = FALSE,
  caption = "Validation Prediction Accuracies Over Bootstrapped Training Fits",
  bold_function = ". == max(., na.rm = TRUE)", bold_margin = 2,
  bold_scheme = c(F, T, F, T, F),
  html_options = list(options = list(dom = "t"))
)

# boxplots
bootstrap_model_errs %>%
  tidyr::pivot_longer(cols = -c(model, bootstrap_id), 
                      names_to = "Metric", values_to = "Value") %>%
  plotBoxplot(x_str = "Value", y_str = "model", horizontal = FALSE) +
  ggplot2::facet_wrap(~ Metric) +
  ggplot2::labs(x = "Method", 
                title = "Validation Prediction Accuracies Over Bootstrapped Training Fits")

3.3 Interpretability

For the models that pass the prediction and stability checks, extract the important features in the predictive models that are stable across both data and model perturbations. Determining the importance of a feature can be method dependent.

Full Model (without stability)

Table

prettyTable(model_results$importance, digits = 2, sigfig = FALSE,
            caption = "Variable Importances")

Plots

plotFeatureImportance(model_results$importance,
                      use_rankings = FALSE,
                      use_facets = TRUE,
                      interactive = FALSE)

plotFeatureImportancePair(model_results$importance,
                          use_rankings = TRUE,
                          interactive = FALSE)

Bootstrapped Model (with stability)

Table

bootstrap_model_imps_summary <- bootstrap_model_imps %>%
  dplyr::group_by(model, variable) %>%
  dplyr::summarise(`Mean Importance` = mean(importance),
                   `Median Importance` = median(importance),
                   `SD Importance` = sd(importance),
                   `Min Importance` = min(importance),
                   `Max Importance` = max(importance))
prettyTable(
  bootstrap_model_imps_summary, 
  digits = 2, sigfig = F, 
  caption = "Summary of variable importances across bootstrapped models"
)

Plots

plotFeatureImportanceStability(bootstrap_model_imps,
                               use_rankings = FALSE,
                               use_facets = FALSE,
                               interactive = FALSE)

4 Main Results

Interpret and summarize the prediction and stability results.

Evaluate pipeline on test data.

Xtrain_final <- dplyr::bind_rows(Xtrain, Xvalid)
ytrain_final <- c(ytrain, yvalid)
if (params$modeling_pkg == "caret") {
  final_model_results <- fitCaret(Xtrain = Xtrain_final, ytrain = ytrain_final, 
                                  Xtest = Xtest, ytest = ytest, 
                                  model_list = model_list, 
                                  tr_control = tr_control)
} else if (params$modeling_pkg == "tidymodels") {
  final_model_results <- fitTidyModels(Xtrain = Xtrain_final, 
                                       ytrain = ytrain_final, 
                                       Xtest = Xtest, ytest = ytest,
                                       model_list = model_list, kfolds = kfolds)
} else if (params$modeling_pkg == "h2o") {
  final_model_results <- fith2o(Xtrain = Xtrain_final, ytrain = ytrain_final,
                                Xtest = Xtest, ytest = ytest,
                                model_list = model_list)
}
#> Growing trees.. Progress: 78%. Estimated remaining time: 8 seconds.
#> Growing trees.. Progress: 84%. Estimated remaining time: 5 seconds.
#> [18:05:22] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:22] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:22] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:22] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:22] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:22] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:22] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:22] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:32] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:32] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:32] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:32] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:32] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:33] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:33] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:33] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:38] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:38] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:38] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:38] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:38] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:38] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:38] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:38] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:44] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:44] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:44] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:44] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:44] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:44] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:44] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:44] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:05:53] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:04] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:10] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:10] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:10] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:10] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:10] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:10] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:10] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:10] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:16] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:16] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:16] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:16] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:16] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:16] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:16] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:16] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:24] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:24] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:24] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:24] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:24] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:24] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:24] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:24] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:35] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:35] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:35] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:35] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:35] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:35] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:35] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:35] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:40] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:40] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:40] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:40] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:40] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:40] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:40] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:40] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:46] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:55] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:55] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:55] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:55] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:55] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:55] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:55] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:06:55] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:05] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:05] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:05] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:05] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:05] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:05] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:05] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:05] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:11] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:11] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:11] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:11] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:11] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:11] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:11] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:11] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:17] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:26] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:26] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:26] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:26] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:26] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:26] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:26] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:26] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:36] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:36] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:36] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:36] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:36] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:36] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:36] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:36] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:42] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:48] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:48] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:48] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:48] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:48] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:48] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:48] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.
#> [18:07:48] WARNING: amalgamation/../src/c_api/c_api.cc:718: `ntree_limit` is deprecated, use `iteration_range` instead.

Summarize test set prediction and/or interpretability results.

# table of accuracy metrics
prettyTable(final_model_results$errors, 
            digits = 2, sigfig = FALSE, rownames = FALSE,
            caption = "Test Prediction Accuracies", 
            html_options = list(options = list(dom = "t")))

5 Post hoc analysis

Move beyond the global prediction accuracy metrics and dive deeper into individual-level predictions for the validation and/or test set, i.e., provide a more “local” analysis.

Examine any points that had poor predictions.
Examine differences between prediction methods.

plotPairs(cbind(`True Repsonses` = ytest, final_model_results$predictions), 
          columns = 1:(ncol(final_model_results$predictions) + 1), 
          title = "Comparison of model predictions")

6 Conclusions

Reiterate main findings, note any caveats, and clearly translate findings/analysis back to the domain problem context.

LS0tCnRpdGxlOiAiUENTIERvY3VtZW50YXRpb24iCmF1dGhvcjogIiIKZGF0ZTogImByIGZvcm1hdChTeXMudGltZSgpLCAnJUIgJWQsICVZJylgIgpoZWFkZXItaW5jbHVkZXM6CiAgICAtIFx1c2VwYWNrYWdle2Zsb2F0fQogICAgLSBcdXNlcGFja2FnZXthbXNtYXRofQogICAgLSBcdXNlcGFja2FnZXtnZW5zeW1ifQogICAgLSBcdXNlcGFja2FnZXt0Y29sb3Jib3h9Cm91dHB1dDoKICBybWRmb3JtYXRzOjptYXRlcmlhbDoKICAgIGZpZ19jYXB0aW9uOiB0cnVlCiAgICBjb2RlX2ZvbGRpbmc6IGhpZGUKICAgIG51bWJlcl9zZWN0aW9uczogdHJ1ZQogICAgdXNlX2Jvb2tkb3duOiB0cnVlCiAgICBmaWdfd2lkdGg6IDEwCiAgICBmaWdfaGVpZ2h0OiA4CiAgICBsaWdodGJveDogdHJ1ZQogICAgY29kZV9kb3dubG9hZDogdHJ1ZQogICAgaW5jbHVkZXM6CiAgICAgIGJlZm9yZV9ib2R5OiBodG1sL3NldHVwLmh0bWwKICBwZGZfZG9jdW1lbnQ6CiAgICBudW1iZXJfc2VjdGlvbnM6IHRydWUKcGFyYW1zOgogICMjIElOUFVUIFBBUkFNRVRFUlMgSEVSRSBPUiBVU0lORyAiS25pdCB3aXRoIFBhcmFtZXRlcnMuLi4iIElOIEFCT1ZFIEtOSVQgTUVOVQogIFhfZmlsZXBhdGg6CiAgICBsYWJlbDogIlggRGF0YSIKICAgIHZhbHVlOiBkYXRhL3RjZ2FfYnJjYV9hcnJheV9kYXRhLnJkcwogICAgaW5wdXQ6IGZpbGUgCiAgeV9maWxlcGF0aDoKICAgIGxhYmVsOiAieSBkYXRhIgogICAgdmFsdWU6IGRhdGEvdGNnYV9icmNhX3N1YnR5cGVzLnJkcwogICAgaW5wdXQ6IGZpbGUKICB0cmFpbl9wcm9wOgogICAgbGFiZWw6ICJUcmFpbmluZyBkYXRhIHByb3BvcnRpb24iCiAgICB2YWx1ZTogMC42CiAgICBpbnB1dDogbnVtZXJpYwogIHZhbGlkX3Byb3A6CiAgICBsYWJlbDogIlZhbGlkYXRpb24gZGF0YSBwcm9wb3J0aW9uIgogICAgdmFsdWU6IDAuMgogICAgaW5wdXQ6IG51bWVyaWMKICB0ZXN0X3Byb3A6CiAgICBsYWJlbDogIlRlc3QgZGF0YSBwcm9wb3J0aW9uIgogICAgdmFsdWU6IDAuMgogICAgaW5wdXQ6IG51bWVyaWMKICBtb2RlbGluZ19wa2c6CiAgICBsYWJlbDogIk1vZGVsaW5nIFBhY2thZ2UiCiAgICB2YWx1ZTogImNhcmV0IgogICAgaW5wdXQ6IHNlbGVjdAogICAgY2hvaWNlczogWyJjYXJldCIsICJoMm8iLCAidGlkeW1vZGVscyJdCiAgc2VlZDoKICAgIGxhYmVsOiAiUmFuZG9tIFNlZWQiCiAgICB2YWx1ZTogMTIzNDUKICAgIGlucHV0OiBudW1lcmljCmNzczogW2Nzcy9jdXN0b21fbWF0ZXJpYWxfcm1kX3RoZW1lLmNzcywgY3NzL2N1c3RvbV9jaGVja2JveC5jc3NdCiMgcnVudGltZTogc2hpbnkKLS0tCgpgYGB7ciBzZXR1cCwgZWNobyA9IEZBTFNFLCBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQpzb3VyY2UoInNjcmlwdHMvcm1kLXNldHVwLlIiLCBsb2NhbCA9IGtuaXRyOjprbml0X2dsb2JhbCgpKQoKIyBsb2FkIGluIHBhY2thZ2VzCmxpYnJhcnkodGlkeXZlcnNlKQpsaWJyYXJ5KHNraW1yKQpsaWJyYXJ5KFIudXRpbHMpCmxpYnJhcnkoZGF0YXNldHMpCmxpYnJhcnkoZ2dwbG90MikKbGlicmFyeShkcGx5cikKbGlicmFyeSh0aWR5cikKbGlicmFyeShyZXNoYXBlMikKZm9yIChmIGluIGxpc3QuZmlsZXMoIlIiLCBwYXR0ZXJuID0gIi5SIiwgZnVsbC5uYW1lcyA9IFRSVUUpKSB7CiAgc291cmNlKGYsIGNoZGlyID0gVCkKfQoKIyBzZXQgc2VlZApzZXQuc2VlZChwYXJhbXMkc2VlZCkKCiMgdmFsaWRhdGUgaW5wdXRzCnZhbGlkYXRlRGF0YVNwbGl0KHBhcmFtcyR0cmFpbl9wcm9wLCBwYXJhbXMkdmFsaWRfcHJvcCwgcGFyYW1zJHRlc3RfcHJvcCkKaWYgKHBhcmFtcyRtb2RlbGluZ19wa2cgIT0gImNhcmV0IikgewogIHN0b3AoIk9ubHkgY2FyZXQgbW9kZWxpbmcgYmFja2VuZCBoYXMgYmVlbiBpbXBsZW1lbnRlZC4iKQp9CgojIGxvYWQgZGF0YSBpbgpYIDwtIGxvYWRGaWxlKHBhcmFtcyRYX2ZpbGVwYXRoKQp5IDwtIGxvYWRGaWxlKHBhcmFtcyR5X2ZpbGVwYXRoKQp2YWxpZGF0ZURhdGEoWCwgeSkKCiMgaW5pdGlhbGl6ZSBjb3VudGVyIGZvciBzdWJjaHVua2lmeQpjaHVua19pZHggPC0gMQpgYGAKCiMgRG9tYWluIHByb2JsZW0gZm9ybXVsYXRpb24KCldoYXQgaXMgdGhlIHJlYWwtd29ybGQgcXVlc3Rpb24/IFRoaXMgY291bGQgYmUgaHlwb3RoZXNpcy1kcml2ZW4gb3IgZGlzY292ZXJ5LWJhc2VkLgoKYGBge2FzaXMsIGhlbHAgPSBUUlVFfQpTb21lIGFkdmljZQpgYGAKCmBgYHthc2lzLCBpbnRlcmFjdGl2ZV90ZXh0ID0gVFJVRX0KCmBgYAoKV2h5IGlzIHRoaXMgcXVlc3Rpb24gaW50ZXJlc3RpbmcgYW5kIGltcG9ydGFudD8gV2hhdCBhcmUgdGhlIGltcGxpY2F0aW9ucyBvZiBiZXR0ZXIgdW5kZXJzdGFuZGluZyB0aGlzIGRhdGE/CgpgYGB7YXNpcywgaW50ZXJhY3RpdmVfdGV4dCA9IFRSVUV9CgpgYGAKCkJyaWVmbHkgZGVzY3JpYmUgYW55IGJhY2tncm91bmQgaW5mb3JtYXRpb24gbmVjZXNzYXJ5IHRvIHVuZGVyc3RhbmQgdGhpcyBwcm9ibGVtLgoKYGBge2FzaXMsIGhlbHAgPSBUUlVFfQpTb21lIGFkdmljZQpgYGAKCmBgYHthc2lzLCBpbnRlcmFjdGl2ZV90ZXh0ID0gVFJVRX0KCmBgYAoKQnJpZWZseSBkZXNjcmliZSBob3cgdGhpcyBxdWVzdGlvbiBjYW4gYmUgYW5zd2VyZWQgaW4gdGhlIGNvbnRleHQgb2YgYSBtb2RlbCBvciBhbmFseXNpcy4KCmBgYHthc2lzLCBpbnRlcmFjdGl2ZV90ZXh0ID0gVFJVRX0KCmBgYAoKT3V0bGluZSB0aGUgcmVzdCBvZiB0aGUgcmVwb3J0L2FuYWx5c2lzLgoKYGBge2FzaXMsIGludGVyYWN0aXZlX3RleHQgPSBUUlVFfQoKYGBgCgojIERhdGEKCldoYXQgaXMgdGhlIGRhdGEgdW5kZXIgaW52ZXN0aWdhdGlvbj8gUHJvdmlkZSBhIGJyaWVmIG92ZXJ2aWV3L2Rlc2NyaXB0aW9uIG9mIHRoZSBkYXRhLgoKYGBge2FzaXMsIGludGVyYWN0aXZlX3RleHQgPSBUUlVFfQoKYGBgCgpEZXNjcmliZSBob3cgeW91ciBkYXRhIGNvbm5lY3RzIHRvIHRoZSBkb21haW4gcHJvYmxlbS4KCmBgYHthc2lzLCBpbnRlcmFjdGl2ZV90ZXh0ID0gVFJVRX0KCmBgYAoKIyMgRGF0YSBDb2xsZWN0aW9uCgpIb3cgd2FzIHRoZSBkYXRhIGNvbGxlY3RlZCBvciBnZW5lcmF0ZWQgKGluY2x1ZGluZyBkZXRhaWxzIG9uIHRoZSBleHBlcmltZW50YWwgZGVzaWduKT8gQmUgYXMgdHJhbnNwYXJlbnQgYXMgcG9zc2libGUgc28gdGhhdCBjb25jbHVzaW9ucyBtYWRlIGZyb20gdGhpcyBkYXRhIGFyZSBub3QgbWlzaW50ZXJwcmV0ZWQgZG93biB0aGUgcm9hZC4KCmBgYHthc2lzLCBpbnRlcmFjdGl2ZV90ZXh0ID0gVFJVRX0KCmBgYAoKRGVzY3JpYmUgYW55IGxpbWl0YXRpb25zIHdoZW4gdXNpbmcgdGhlIGRhdGEgdG8gYW5zd2VyIHRoZSBkb21haW4gcHJvYmxlbSBvZiBpbnRlcmVzdC4KCmBgYHthc2lzLCBpbnRlcmFjdGl2ZV90ZXh0ID0gVFJVRX0KCmBgYAoKV2hlcmUgaXMgdGhlIGRhdGEgc3RvcmVkLCBhbmQgaG93IGNhbiBpdCBiZSBhY2Nlc3NlZCBieSBvdGhlcnMgKGlmIGFwcGxpY2FibGUpPwoKYGBge2FzaXMsIGludGVyYWN0aXZlX3RleHQgPSBUUlVFfQoKYGBgCgojIyBEYXRhIFNwbGl0dGluZwoKVE9ETzogYWRkIGFkdmljZSBmb3IgcG9zc2libGUgZGF0YSBzcGxpdHMsIEFLIGdldHRpbmcgbmljZSBmaWd1cmUgdG9nZXRoZXIKCkRlY2lkZSBvbiB0aGUgcHJvcG9ydGlvbiBvZiBkYXRhIGluIGVhY2ggc3BsaXQuCgpEZWNpZGUgb24gdGhlICJob3ciIHRvIHNwbGl0IHRoZSBkYXRhIChlLmcuLCByYW5kb20gc2FtcGxpbmcsIHN0cmF0aWZpZWQgc2FtcGxpbmcsIGV0Yy4pLCBhbmQgZXhwbGFpbiB3aHkgdGhpcyBpcyBhIHJlYXNvbmFibGUgd2F5IHRvIHNwbGl0IHRoZSBkYXRhLgoKYGBge2FzaXMsIGludGVyYWN0aXZlX3RleHQgPSBUUlVFfQoKYGBgCgpTcGxpdCB0aGUgZGF0YSBpbnRvIGEgdHJhaW5pbmcsIHZhbGlkYXRpb24sIGFuZCB0ZXN0IHNldC4KCmBgYHtyIHNwbGl0LWRhdGF9CmRhdGFfc3BsaXQgPC0gZGF0YVNwbGl0KFggPSBYLCB5ID0geSwgc3RyYXRpZmllZF9ieSA9IHksCiAgICAgICAgICAgICAgICAgICAgICAgIHRyYWluX3Byb3AgPSBwYXJhbXMkdHJhaW5fcHJvcCwgCiAgICAgICAgICAgICAgICAgICAgICAgIHZhbGlkX3Byb3AgPSBwYXJhbXMkdmFsaWRfcHJvcCwgCiAgICAgICAgICAgICAgICAgICAgICAgIHRlc3RfcHJvcCA9IHBhcmFtcyR0ZXN0X3Byb3ApClh0cmFpbiA8LSBkYXRhX3NwbGl0JFgkdHJhaW4KWHZhbGlkIDwtIGRhdGFfc3BsaXQkWCR2YWxpZGF0ZQpYdGVzdCA8LSBkYXRhX3NwbGl0JFgkdGVzdAp5dHJhaW4gPC0gZGF0YV9zcGxpdCR5JHRyYWluCnl2YWxpZCA8LSBkYXRhX3NwbGl0JHkkdmFsaWRhdGUKeXRlc3QgPC0gZGF0YV9zcGxpdCR5JHRlc3QKYGBgCgpQcm92aWRlIHN1bW1hcnkgc3RhdGlzdGljcyBhbmQvb3IgZmlndXJlcyBvZiB0aGUgdGhyZWUgZGF0YSBzZXRzIHRvIGlsbHVzdHJhdGUgaG93IHNpbWlsYXIgKG9yIGRpZmZlcmVudCkgdGhleSBhcmUuCgpgYGB7YXNpcywgaW50ZXJhY3RpdmVfdGV4dCA9IFRSVUV9CgpgYGAKCiMjIyBEYXRhIFNwbGl0dGluZyBPdmVydmlldyB7LnRhYnNldCAuY3VzdG9tLXRhYnN9CgojIyMjIFggRGF0YSBTcGxpdCB7LnVubnVtYmVyZWR9CgpgYGB7ciBYLWRhdGEtc3BsaXQsIGFkZF9uZXdfbGluZSA9IFRSVUV9CnBsb3REYXRhU3BsaXQoWHRyYWluLCBYdmFsaWQsIFh0ZXN0LCAKICAgICAgICAgICAgICB4bGFiID0gIlgiLCB0aXRsZSA9ICJPdmVyYWxsIFggRGlzdHJpYnV0aW9uIikKYGBgCgojIyMjIFkgRGF0YSBTcGxpdCB7LnVubnVtYmVyZWR9CgpgYGB7ciB5LWRhdGEtc3BsaXQsIGFkZF9uZXdfbGluZSA9IFRSVUV9CnBsb3REYXRhU3BsaXQoeXRyYWluLCB5dmFsaWQsIHl0ZXN0LCAKICAgICAgICAgICAgICB4bGFiID0gInkiLCB0aXRsZSA9ICJPdmVyYWxsIHkgRGlzdHJpYnV0aW9uIikKYGBgCgojIyBEYXRhIENsZWFuaW5nIGFuZCBQcmVwcm9jZXNzaW5nCgpXaGF0IHN0ZXBzIHdlcmUgdGFrZW4gdG8gY2xlYW4gdGhlIGRhdGE/IE1vcmUgaW1wb3J0YW50bHksIHdoeSB3YXMgdGhlIGRhdGEgY2xlYW5lZCBpbiB0aGlzIHdheT8KCkRpc2N1c3MgYWxsIGluY29uc2lzdGVuY2llcywgcHJvYmxlbXMsIG9kZGl0aWVzIGluIHRoZSBkYXRhIChlLmcuLCBtaXNzaW5nIGRhdGEsIGVycm9ycyBpbiBkYXRhLCBvdXRsaWVycywgZXRjLikuCgpSZWNvcmQgeW91ciBwcmVwcm9jZXNzaW5nIHN0ZXBzIGluIGEgd2F5IHN1Y2ggdGhhdCBpZiBzb21lb25lIGVsc2Ugd2VyZSB0byByZXByb2R1Y2UgeW91ciBhbmFseXNpcywgdGhleSBjb3VsZCBlYXNpbHkgcmVwbGljYXRlIGFuZCB1bmRlcnN0YW5kIHlvdXIgc3RlcHMuCgpJdCBjYW4gYmUgaGVscGZ1bCB0byBpbmNsdWRlIHJlbGV2YW50IHBsb3RzIHRoYXQgZXhwbGFpbi9qdXN0aWZ5IHRoZSBjaG9pY2VzIHRoYXQgd2VyZSBtYWRlIHdoZW4gY2xlYW5pbmcgdGhlIGRhdGEuCgpJZiBtb3JlIHRoYW4gb25lIHByZXByb2Nlc3NpbmcgcGlwZWxpbmUgaXMgcmVhc29uYWJsZSwgZXhhbWluZSB0aGUgaW1wYWN0cyBvZiB0aGVzZSBhbHRlcm5hdGl2ZSBwcmVwcm9jZXNzaW5nIHBpcGVsaW5lcyBvbiB0aGUgZmluYWwgZGF0YSByZXN1bHRzLgoKQWdhaW4sIGJlIGFzIHRyYW5zcGFyZW50IGFzIHBvc3NpYmxlLiBUaGlzIGFsbG93cyBvdGhlcnMgdG8gbWFrZSB0aGVpciBvd24gZWR1Y2F0ZWQgZGVjaXNpb25zIG9uIGhvdyBiZXN0IHRvIHByZXByb2Nlc3MgdGhlIGRhdGEuCgpgYGB7YXNpcywgaW50ZXJhY3RpdmVfdGV4dCA9IFRSVUV9CkdpdmVuIHRoaXMgZXhhbXBsZSBUQ0dBIEJSQ0EgZGF0YSBzZXQsIHdlIGZpcnN0IHByZXByb2Nlc3MgdGhlIGRhdGEgYnkgcmVtb3ZpbmcgY29uc3RhbnQgb3IgZHVwbGljYXRlZCBjb2x1bW5zLiBUaGVuIHNpbmNlIHRoZSBhcnJheSBkYXRhIGlzIGhpZ2hseSByaWdodCBza2V3ZWQsIHdlIHdpbGwgbG9nLXRyYW5zZm9ybSAoaS5lLiwgbG9nKHggKyAxKSkgdGhlIGRhdGEuIEZpbmFsbHksIHRvIGtlZXAgdGhpcyBleGFtcGxlIHRlbXBsYXRlIHJlbGF0aXZlbHkgcXVpY2sgdG8gcnVuLCB3ZSB3aWxsIG9ubHkga2VlcCB0aGUgMTAwMCBmZWF0dXJlcyB3aXRoIHRoZSBoaWdoZXN0IHZhcmlhbmNlLgpgYGAKCmBgYHtyIHByZXByb2Nlc3MtZGF0YX0KIyMgRE8gREFUQSBDTEVBTklORyAvIFBSRS1QUk9DRVNTSU5HIEhFUkUgClh0cmFpbiA8LSBsb2coWHRyYWluICsgMSkgJT4lCiAgcmVtb3ZlQ29uc3RhbnRDb2xzKHZlcmJvc2UgPSAxKSAlPiUKICByZW1vdmVEdXBsaWNhdGVDb2xzKHZlcmJvc2UgPSAxKSAlPiUKICBmaWx0ZXJDb2xzQnlWYXIobWF4X3AgPSAxMDAwKQpYdmFsaWQgPC0gbG9nKFh2YWxpZCArIDEpWywgY29sbmFtZXMoWHRyYWluKV0KWHRlc3QgPC0gbG9nKFh0ZXN0ICsgMSlbLCBjb2xuYW1lcyhYdHJhaW4pXQpgYGAKCgojIyBEYXRhIEV4cGxvcmF0aW9uCgpUT0RPOiBBZGQgZHJhZyBhbmQgZHJvcCBmZWF0dXJlIGluIHNoaW55IHZlcnNpb24gZm9yIG90aGVyIGltYWdlcwoKVGhlIG1haW4gZ29hbCBvZiB0aGlzIHNlY3Rpb24gaXMgdG8gZ2l2ZSB0aGUgcmVhZGVyIGEgZmVlbCBmb3Igd2hhdCB0aGUgZGF0YSAibG9va3MgbGlrZSIgYXQgYSBiYXNpYyBsZXZlbC4KClByb3ZpZGUgcGxvdHMgdGhhdCBzdW1tYXJpemUgdGhlIGRhdGEgYW5kIHBlcmhhcHMgZXZlbiBwbG90cyB0aGF0IGNvbnZleSBzb21lIHNtYWxsZXIgZmluZGluZ3Mgd2hpY2ggdWx0aW1hdGVseSBtb3RpdmF0ZSB0aGUgbWFpbiBmaW5kaW5ncy4KClByb3ZpZGUgYWRkaXRpb25hbCBwbG90cyByZXByZXNlbnRpbmcgcmVtYWluaW5nIG9kZGl0aWVzIGFmdGVyIHByZS1wcm9jZXNzaW5nIGlmIGFwcGxpY2FibGUuCgpBZGQgc3VtbWFyeSBzdGF0aXN0aWNzIGluIGFjY29tcGFueWluZyB0YWJsZXMgKG9yIGluIGZpZ3VyZXMpIGZvciBxdWljayBjb21wYXJpc29ucy4KCmBgYHthc2lzLCBpbnRlcmFjdGl2ZV90ZXh0ID0gVFJVRX0KCmBgYAoKIyMjIERhdGEgT3ZlcnZpZXcgey50YWJzZXQgLmN1c3RvbS10YWJzfQoKYGBge3IgZGF0YS1kaW1zLCBlY2hvID0gRkFMU0V9CiMgc3VtbWFyeSBvZiBkYXRhIGRpbWVuc2lvbnMKZGF0YURpbWVuc2lvbnMoWHRyYWluID0gWHRyYWluLCBYdmFsaWQgPSBYdmFsaWQsIFh0ZXN0ID0gWHRlc3QpCmBgYAoKYGBge3IgZGF0YS10eXBlcywgcmVzdWx0cz0iYXNpcyIsIGFkZF9uZXdfbGluZSA9IFRSVUV9CiMgc3VtbWFyeSBvZiB0eXBlcyBvZiBmZWF0dXJlcyBpbiAoWCwgeSkgZGF0YQpkYXRhVHlwZXMoWCA9IFh0cmFpbiwgeSA9IHl0cmFpbikKYGBgCgojIyMjIFN1bW1hcnkgVGFibGVzIHsudW5udW1iZXJlZH0KCmBgYHtyIGRhdGEtc3VtbWFyeS10YWJsZSwgcmVzdWx0cz0iYXNpcyIsIGFkZF9uZXdfbGluZSA9IFRSVUV9CiMgYnJvYWQgYXJyYXkgb2Ygc3VtbWFyeSBzdGF0aXN0aWNzIGZvciBmZWF0dXJlcyBpbiB0cmFpbmluZyAoWCwgeSkKdGFiX2xzIDwtIGRhdGFTdW1tYXJ5KFggPSBYdHJhaW4sIHkgPSB5dHJhaW4pCmZvciAoZHR5cGUgaW4gbmFtZXModGFiX2xzKSkgeyAgIyBncm91cGVkIGJ5IGRhdGEgdHlwZQogIHNpbUNoZWY6OjpzdWJjaHVua2lmeSh0YWJfbHNbW2R0eXBlXV0sIGkgPSBjaHVua19pZHgsIAogICAgICAgICAgICAgICAgICAgICAgICBvdGhlcl9hcmdzID0gInJlc3VsdHM9J2FzaXMnIikKICBjaHVua19pZHggPC0gY2h1bmtfaWR4ICsgMQp9CmBgYAoKIyMjIyBYIERpc3RyaWJ1dGlvbiB7LnVubnVtYmVyZWR9CgpgYGB7ciB4LWRpc3QtcGxvdCwgYWRkX25ld19saW5lID0gVFJVRX0KIyBwbG90IFggZGlzdHJpYnV0aW9uCnBsb3REYXRhRGlzdHJpYnV0aW9uKGRhdGEgPSBYdHJhaW4sIAogICAgICAgICAgICAgICAgICAgICB4bGFiID0gIlgiLCB0aXRsZSA9ICJUcmFpbmluZyBYIERpc3RyaWJ1dGlvbiIpCmBgYAoKIyMjIyBZIERpc3RyaWJ1dGlvbiB7LnVubnVtYmVyZWR9CgpgYGB7ciB5LWRpc3QtcGxvdCwgYWRkX25ld19saW5lID0gVFJVRX0KIyBwbG90IHkgZGlzdHJpYnV0aW9uCnBsb3REYXRhRGlzdHJpYnV0aW9uKGRhdGEgPSB5dHJhaW4sIAogICAgICAgICAgICAgICAgICAgICB4bGFiID0gInkiLCB0aXRsZSA9ICJUcmFpbmluZyB5IERpc3RyaWJ1dGlvbiIpCmBgYAoKIyMjIyBEYXRhIEhlYXRtYXAgey51bm51bWJlcmVkfQoKYGBge3IgZGF0YS1oZWF0bWFwLCBhZGRfbmV3X2xpbmUgPSBUUlVFfQojIChjbHVzdGVyZWQpIGhlYXRtYXAgb2YgKFgsIHkpIGRhdGEKcGxvdERhdGFIZWF0bWFwKFggPSBYdHJhaW4sIHkgPSB5dHJhaW4sIGNsdXN0X3Jvd3MgPSBUUlVFLCBjbHVzdF9jb2xzID0gVFJVRSwKICAgICAgICAgICAgICAgIHNob3dfeXRleHQgPSBGQUxTRSwgeF90ZXh0X2FuZ2xlID0gVFJVRSkKYGBgCgojIyMjIEZlYXR1cmUgQ29ycmVsYXRpb24gey51bm51bWJlcmVkfQoKYGBge3IgZmVhdHVyZS1jb3ItaGVhdG1hcCwgYWRkX25ld19saW5lID0gVFJVRX0KIyAoY2x1c3RlcmVkKSBjb3JyZWxhdGlvbiBoZWF0bWFwCnBsb3RDb3JIZWF0bWFwKFggPSBYdHJhaW4sIGNvcl90eXBlID0gInBlYXJzb24iLCBjbHVzdCA9IFRSVUUsCiAgICAgICAgICAgICAgIHhfdGV4dF9hbmdsZSA9IFRSVUUpICsKICBnZ3Bsb3QyOjpsYWJzKHggPSAiRmVhdHVyZXMiLCB5ID0gIkZlYXR1cmVzIiwgZmlsbCA9ICJDb3IuIikKYGBgCgojIyMjIEZlYXR1cmUgUGFpciBQbG90cyB7LnVubnVtYmVyZWR9CgpgYGB7ciBmZWF0dXJlLXBhaXItcGxvdCwgYWRkX25ld19saW5lID0gVFJVRX0KIyBmZWF0dXJlIHBhaXIgcGxvdCAoZm9yIGEgc3Vic2V0IG9mIGZlYXR1cmVzKQprZWVwX2ZlYXR1cmVzIDwtIHNvcnQoCiAgc2FtcGxlKDE6bmNvbChYdHJhaW4pLCBtaW4obmNvbChYdHJhaW4pLCA2KSwgcmVwbGFjZSA9IEZBTFNFKQopCnBsb3RQYWlycyhkYXRhID0gWHRyYWluLCBjb2x1bW5zID0ga2VlcF9mZWF0dXJlcywgCiAgICAgICAgICBjb2xvciA9IHl0cmFpbiwgY29sb3JfbGFiZWwgPSAieSIpCmBgYAoKIyMjIyBNYXJnaW5hbCBBc3NvY2lhdGlvbnMgey51bm51bWJlcmVkfQoKYGBge3IgbWFyZ2luYWwtYXNzb2MtcGxvdCwgYWRkX25ld19saW5lID0gVFJVRX0KIyBtYXJnaW5hbCBhc3NvY2lhdGlvbiBwbG90cyAoZm9yIGEgc3Vic2V0IG9mIGZlYXR1cmVzKQpjYXJldDo6ZmVhdHVyZVBsb3QoeCA9IFh0cmFpblssIGtlZXBfZmVhdHVyZXNdLAogICAgICAgICAgICAgICAgICAgeSA9IHl0cmFpbiwKICAgICAgICAgICAgICAgICAgIHBsb3QgPSBpZiAoaXMuZmFjdG9yKHl0cmFpbikpICJib3giIGVsc2UgInNjYXR0ZXIiLAogICAgICAgICAgICAgICAgICAgc2NhbGVzID0gbGlzdCh4ID0gbGlzdChyZWxhdGlvbiA9ICJmcmVlIiksIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB5ID0gbGlzdChyZWxhdGlvbiA9ICJmcmVlIikpKQpgYGAKCiMjIyMgUENBIHsudW5udW1iZXJlZH0KCmBgYHtyIHBjYS1wbG90LCBhZGRfbmV3X2xpbmUgPSBUUlVFfQojIHBjYSBwbG90CnBsb3RQQ0EoWCA9IFh0cmFpbiwgbnBjcyA9IDMsIGNvbG9yID0geXRyYWluLCBjb2xvcl9sYWJlbCA9ICJ5IiwKICAgICAgICBjZW50ZXIgPSBUUlVFLCBzY2FsZSA9IEZBTFNFKSRwbG90CmBgYAoKIyBQcmVkaWN0aW9uIE1vZGVsaW5nCgpUT0RPOiBhZGQgYWR2aWNlIG9uIHdoaWNoIG1vZGVscyB0byBzZWxlY3QgYW5kIHdoeQoKRGlzY3VzcyB0aGUgcHJlZGljdGlvbiBtZXRob2RzIHVuZGVyIGNvbnNpZGVyYXRpb24sIGFuZCBleHBsYWluIHdoeSB0aGVzZSBtZXRob2RzIHdlcmUgY2hvc2VuLgoKYGBge2FzaXMsIGludGVyYWN0aXZlX3RleHQgPSBUUlVFfQoKYGBgCgpEaXNjdXNzIHRoZSBhY2N1cmFjeSBtZXRyaWNzIHVuZGVyIGNvbnNpZGVyYXRpb24sIGFuZCBleHBsYWluIHdoeSB0aGVzZSBtZXRyaWNzIHdlcmUgY2hvc2VuLgoKYGBge2FzaXMsIGludGVyYWN0aXZlX3RleHQgPSBUUlVFfQoKYGBgCgpOb3RlOiB0aGVyZSBzaG91bGQgYmUgbXVsdGlwbGUgbWV0aG9kcyBhbmQgbWV0cmljcyB1bmRlciBjb25zaWRlcmF0aW9uIHRvIHBhaW50IGEgbW9yZSBob2xpc3RpYyBwaWN0dXJlIG9mIHRoZSBkYXRhLiBBdCBsZWFzdCBvbmUgbWV0aG9kIHNob3VsZCBiZSBhIGJhc2VsaW5lLCBjb21tb24gYXBwcm9hY2ggdGhhdCBtYXkgbm90IGJlIG9wdGltYWwgZm9yIHRoZSBwcm9ibGVtIHNldHRpbmcsIGJ1dCBzZXJ2ZXMgYXMgYSBoZWxwZnVsIGNvbXBhcmlzb24uCgojIyBQcmVkaWN0aW9uIGNoZWNrCgpDYXJyeSBvdXQgdGhlIHByZWRpY3Rpb24gcGlwZWxpbmUsIG91dGxpbmVkIGFib3ZlLgoKLSAgIEZpdCBwcmVkaWN0aW9uIG1ldGhvZHMgb24gdHJhaW5pbmcgZGF0YS4KLSAgIEV2YWx1YXRlIHByZWRpY3Rpb24gbWV0aG9kcyBvbiB2YWxpZGF0aW9uIGRhdGEuCi0gICBDb21wYXJlIHJlc3VsdHMsIGFuZCBmaWx0ZXIgb3V0IHBvb3IgbW9kZWxzLgoKYGBge2FzaXMsIGludGVyYWN0aXZlX3RleHQgPSBUUlVFfQoKYGBgCgpgYGB7ciBjYXJldC1maXRzLCBldmFsID0gcGFyYW1zJG1vZGVsaW5nX3BrZyA9PSAiY2FyZXQiLCBlY2hvID0gcGFyYW1zJG1vZGVsaW5nX3BrZyA9PSAiY2FyZXQifQojIyBJRiBVU0lORyBDQVJFVCBNT0RFTElORyBCQUNLRU5ELCBDSE9PU0UgTUVUSE9EUyBBTkQgVFJBSU5JTkcgQ09OVFJPTFMgSEVSRQp0cl9jb250cm9sIDwtIGNhcmV0Ojp0cmFpbkNvbnRyb2woCiAgbWV0aG9kID0gImN2IiwKICBudW1iZXIgPSA1LAogIGNsYXNzUHJvYnMgPSBpZiAoaXMuZmFjdG9yKHl0cmFpbikpIFRSVUUgZWxzZSBGQUxTRSwKICBzdW1tYXJ5RnVuY3Rpb24gPSBjYXJldDo6ZGVmYXVsdFN1bW1hcnksCiAgYWxsb3dQYXJhbGxlbCA9IEZBTFNFLAogIHZlcmJvc2VJdGVyID0gRkFMU0UKKQoKbW9kZWxfbGlzdCA8LSBsaXN0KAogIHJhbmdlciA9IGxpc3QodHVuZUdyaWQgPSBleHBhbmQuZ3JpZChtdHJ5ID0gc2VxKHNxcnQobmNvbChYdHJhaW4pKSwgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbmNvbChYdHJhaW4pIC8gMywKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBsZW5ndGgub3V0ID0gMyksCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHNwbGl0cnVsZSA9ICJnaW5pIiwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbWluLm5vZGUuc2l6ZSA9IDEpLAogICAgICAgICAgICAgICAgaW1wb3J0YW5jZSA9ICJpbXB1cml0eSIsCiAgICAgICAgICAgICAgICBudW0udGhyZWFkcyA9IDEpLAogIHhnYlRyZWUgPSBsaXN0KHR1bmVHcmlkID0gZXhwYW5kLmdyaWQobnJvdW5kcyA9IGMoMTAsIDI1LCA1MCwgMTAwLCAxNTApLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbWF4X2RlcHRoID0gYygzLCA2KSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNvbHNhbXBsZV9ieXRyZWUgPSAwLjMzLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZXRhID0gYygwLjEsIDAuMyksCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBnYW1tYSA9IDAsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBtaW5fY2hpbGRfd2VpZ2h0ID0gMSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHN1YnNhbXBsZSA9IDAuNiksCiAgICAgICAgICAgICAgICAgbnRocmVhZCA9IDEpCikKCnN0YXJ0X3RpbWUgPC0gU3lzLnRpbWUoKQptb2RlbF9yZXN1bHRzIDwtIGZpdENhcmV0KFh0cmFpbiA9IFh0cmFpbiwgeXRyYWluID0geXRyYWluLCAKICAgICAgICAgICAgICAgICAgICAgICAgICBYdGVzdCA9IFh2YWxpZCwgeXRlc3QgPSB5dmFsaWQsIAogICAgICAgICAgICAgICAgICAgICAgICAgIG1vZGVsX2xpc3QgPSBtb2RlbF9saXN0LCB0cl9jb250cm9sID0gdHJfY29udHJvbCkKY2F0KHNwcmludGYoIkZpdHRpbmcgdGltZSB0YWtlbjogJWYgbWludXRlcyIsIAogICAgICAgICAgICBkaWZmdGltZShTeXMudGltZSgpLCBzdGFydF90aW1lLCB1bml0cyA9ICJtaW5zIikpKQpgYGAKCmBgYHtyIGgyby1maXRzLCBldmFsID0gcGFyYW1zJG1vZGVsaW5nX3BrZyA9PSAiaDJvIiwgZWNobyA9IHBhcmFtcyRtb2RlbGluZ19wa2cgPT0gImgybyJ9CiMjIElGIFVTSU5HIEgyTyBNT0RFTElORyBCQUNLRU5ELCBDSE9PU0UgTUVUSE9EUyBBTkQgVFJBSU5JTkcgQ09OVFJPTFMgSEVSRQptb2RlbF9saXN0IDwtIGxpc3QocmFuZG9tRm9yZXN0ID0gbGlzdChudHJlZXMgPSA1MDApLCAKICAgICAgICAgICAgICAgICAgIHhnYm9vc3QgPSBsaXN0KCkpCgpzdGFydF90aW1lIDwtIFN5cy50aW1lKCkKbW9kZWxfcmVzdWx0cyA8LSBmaXRoMm8oWHRyYWluID0gWHRyYWluLCB5dHJhaW4gPSB5dHJhaW4sCiAgICAgICAgICAgICAgICAgICAgICAgIFh0ZXN0ID0gWHZhbGlkLCB5dGVzdCA9IHl2YWxpZCwKICAgICAgICAgICAgICAgICAgICAgICAgbW9kZWxfbGlzdCA9IG1vZGVsX2xpc3QpCmNhdChzcHJpbnRmKCJGaXR0aW5nIHRpbWUgdGFrZW46ICVmIG1pbnV0ZXMiLCAKICAgICAgICAgICAgZGlmZnRpbWUoU3lzLnRpbWUoKSwgc3RhcnRfdGltZSwgdW5pdHMgPSAibWlucyIpKSkKYGBgCgpgYGB7ciB0aWR5bW9kZWxzLWZpdHMsIGV2YWwgPSBwYXJhbXMkbW9kZWxpbmdfcGtnID09ICJ0aWR5bW9kZWxzIiwgZWNobyA9IHBhcmFtcyRtb2RlbGluZ19wa2cgPT0gInRpZHltb2RlbHMifQojIyBJRiBVU0lORyBUSURZTU9ERUxTIEJBQ0tFTkQsIENIT09TRSBNRVRIT0RTIEFORCBUUkFJTklORyBDT05UUk9MUyBIRVJFCnJmX21vZGVsIDwtIHBhcnNuaXA6OnJhbmRfZm9yZXN0KCkgJT4lCiAgcGFyc25pcDo6c2V0X2FyZ3MobXRyeSA9IHR1bmU6OnR1bmUoKSkgJT4lCiAgcGFyc25pcDo6c2V0X2VuZ2luZSgicmFuZ2VyIiwgaW1wb3J0YW5jZSA9ICJpbXB1cml0eSIpICU+JQogIHBhcnNuaXA6OnNldF9tb2RlKCJjbGFzc2lmaWNhdGlvbiIpCnJmX2dyaWQgPC0gdGlkeXI6OmNyb3NzaW5nKG10cnkgPSAxOjQpCgpzdm1fbW9kZWwgPC0gcGFyc25pcDo6c3ZtX3JiZigpICU+JQogIHBhcnNuaXA6OnNldF9lbmdpbmUoImtlcm5sYWIiKSAlPiUKICBwYXJzbmlwOjpzZXRfbW9kZSgiY2xhc3NpZmljYXRpb24iKQoKa25uX21vZGVsIDwtIHBhcnNuaXA6Om5lYXJlc3RfbmVpZ2hib3IoKSAlPiUKICAgcGFyc25pcDo6c2V0X2FyZ3MobmVpZ2hib3JzID0gdHVuZSgpLCB3ZWlnaHRfZnVuYyA9IHR1bmUoKSkgJT4lIAogICBwYXJzbmlwOjpzZXRfZW5naW5lKCJra25uIikgJT4lIAogICBwYXJzbmlwOjpzZXRfbW9kZSgiY2xhc3NpZmljYXRpb24iKQoKbW9kZWxfbGlzdCA8LSBsaXN0KFJGID0gbGlzdChtb2RlbCA9IHJmX21vZGVsLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgIGdyaWQgPSByZl9ncmlkKSwgCiAgICAgICAgICAgICAgICAgICBTVk0gPSBsaXN0KG1vZGVsID0gc3ZtX21vZGVsLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICBncmlkID0gTlVMTCksIAogICAgICAgICAgICAgICAgICAgS05OID0gbGlzdChtb2RlbCA9IGtubl9tb2RlbCwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZ3JpZCA9IDQpKQprZm9sZHMgPC0gNQoKc3RhcnRfdGltZSA8LSBTeXMudGltZSgpCm1vZGVsX3Jlc3VsdHMgPC0gZml0VGlkeU1vZGVscyhYdHJhaW4gPSBYdHJhaW4sIHl0cmFpbiA9IHl0cmFpbiwgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBYdGVzdCA9IFh2YWxpZCwgeXRlc3QgPSB5dmFsaWQsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBtb2RlbF9saXN0ID0gbW9kZWxfbGlzdCwga2ZvbGRzID0ga2ZvbGRzKQpjYXQoc3ByaW50ZigiRml0dGluZyB0aW1lIHRha2VuOiAlZiBtaW51dGVzIiwgCiAgICAgICAgICAgIGRpZmZ0aW1lKFN5cy50aW1lKCksIHN0YXJ0X3RpbWUsIHVuaXRzID0gIm1pbnMiKSkpCmBgYAoKYGBge3IgcHJlZGljdGlvbi1hY2N1cmFjeSwgcmVzdWx0cyA9ICJhc2lzIn0KIyB0YWJsZSBvZiBhY2N1cmFjeSBtZXRyaWNzCnByZXR0eVRhYmxlKG1vZGVsX3Jlc3VsdHMkZXJyb3JzLAogICAgICAgICAgICBkaWdpdHMgPSAyLCBzaWdmaWcgPSBGQUxTRSwgcm93bmFtZXMgPSBGQUxTRSwKICAgICAgICAgICAgY2FwdGlvbiA9ICJWYWxpZGF0aW9uIFByZWRpY3Rpb24gQWNjdXJhY2llcyIsIAogICAgICAgICAgICBodG1sX29wdGlvbnMgPSBsaXN0KG9wdGlvbnMgPSBsaXN0KGRvbSA9ICJ0IikpKQpgYGAKCiMjIFN0YWJpbGl0eSBjaGVjawoKVGFraW5nIHRoZSBwcmVkaWN0aW9uIG1ldGhvZHMgdGhhdCBwYXNzIHRoZSBwcmVkaWN0aW9uIGNoZWNrLCBwZXJmb3JtIHN0YWJpbGl0eSBhbmFseXNpcy4KCi0gICBTcGVjaWZ5IGFuZCBqdXN0aWZ5IHRoZSBhcHByb3ByaWF0ZSBkYXRhIHBlcnR1cmJhdGlvbihzKS4KLSAgIFJlLWZpdCB0aGUgcHJlZGljdGlvbiBtZXRob2RzIG9uIHRoZXNlIHBlcnR1cmJlZCBkYXRhIHNldHMuCi0gICBFdmFsdWF0ZSBwcmVkaWN0aW9uIG1ldGhvZHMgb24gdmFsaWRhdGlvbiBkYXRhLgotICAgQXNzZXNzIHN0YWJpbGl0eSBhY3Jvc3MgdGhlIGRhdGEgcGVydHVyYmF0aW9ucyBhcyB3ZWxsIGFzIGFjcm9zcyB0aGUgdmFyaW91cyBtZXRob2RzLgotICAgRmlsdGVyIG91dCBwb29yIG1vZGVscyB3aGVyZSBuZWNlc3NhcnkgYW5kIGludGVycHJldCBzdGFiaWxpdHkgcmVzdWx0cy4KCmBgYHthc2lzLCBpbnRlcmFjdGl2ZV90ZXh0ID0gVFJVRX0KCmBgYAoKVE9ETzogQWRkIHJlc3VsdHMgZm9yIGgyMCwgdHVuaW5nIG5lZWRzIHRvIGJlIGZpeGVkIGZvciBoMjAgKyB0aWR5bW9kZWxzIDwhLS0gVE9ETzogQW5hIC0gUHJvdmlkZSBzb21lIGV4YW1wbGUgY29kZSBoZXJlIChmb3IgYm90aCBmaXR0aW5nIGFuZCB2aXN1YWxpemluZyByZXN1bHRzKSBzbyB0aGF0IHRoZSBwcmFjdGl0aW9uZXIgY2FuIGVhc2lseSBpbnB1dCB0aGVpciBkYXRhIGFuZCBtb2RlbHMuIEFkZCBzb21ldGhpbmcgbGlrZSBKYW1lcycgc2xpZGUuIEEgZmV3IHBpY3R1cmVzIHdpdGggcG9zc2libGUgZGF0YSBwZXJ0dXJiYXRpb24gc2NoZW1lcyAoc2VwYXJhdGUgZnJvbSBwYXJhbWV0ZXIgdHVuaW5nKS4gRGF0YSBzcGxpdHRpbmcgdnMgc2FtcGxpbmcgb2Ygb2JzZXJ2YXRpb25zLiBDcm9zcy12YWxpZGF0aW9uLWlzaCBzY2hlbWUgYW5kIGZpeGVkIHRyYWluaW5nL3ZhbGlkYXRpb24gc2V0dXAgKGJvb3RzdHJhcHBpbmcsIHN1Yi1zYW1wbGluZywgc3RyYXRpZmllZC1zYW1wbGluZykuIEFkZCBwYXJhbWV0ZXIgdG8gaW5jbHVkZS9leGNsdWRlIGNlcnRhaW4gY29kZSBjaHVua3MuIC0tPgoKYGBge3Igc3RhYmlsaXR5LWNoZWNrfQpuX3JlcHMgPC0gMiAjIGluY3JlYXNlIGZvciBiZXR0ZXIgc3RhYmlsaXR5IG1lYXN1cmVzIHdoZW4gbm90IHRlc3RpbmcgY29kZQoKIyBwIDwtIHByb2dyZXNzcjo6cHJvZ3Jlc3NvcihzdGVwcyA9IG5fcmVwcykKIyBmdXR1cmU6OnBsYW4obXVsdGlzZXNzaW9uLCB3b3JrZXJzID0gbWluKG5fcmVwcywgcGFyYWxsZWw6OmRldGVjdENvcmVzKCkgLSAxKSkKYm9vdHN0cmFwX21vZGVsX3Jlc3VsdHMgPC0gZnV0dXJlLmFwcGx5OjpmdXR1cmVfcmVwbGljYXRlKAogIG4gPSBuX3JlcHMsCiAgZXhwciA9IHsKICAgIGJvb3RzdHJhcCA8LSBzYW1wbGUoMTpucm93KFh0cmFpbiksIG5yb3coWHRyYWluKSwgcmVwbGFjZSA9IFRSVUUpCiAgICBYdHJhaW5fYiA8LSBYdHJhaW5bYm9vdHN0cmFwLCBdCiAgICB5dHJhaW5fYiA8LSB5dHJhaW5bYm9vdHN0cmFwXQogICAgCiAgICBpZiAocGFyYW1zJG1vZGVsaW5nX3BrZyA9PSAiY2FyZXQiKSB7CiAgICAgIG1vZGVsX3Jlc3VsdHNfYiA8LSBmaXRDYXJldChYdHJhaW4gPSBYdHJhaW5fYiwgeXRyYWluID0geXRyYWluX2IsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgWHRlc3QgPSBYdmFsaWQsIHl0ZXN0ID0geXZhbGlkLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIG1vZGVsX2xpc3QgPSBtb2RlbF9saXN0LCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHRyX2NvbnRyb2wgPSB0cl9jb250cm9sKQogICAgfSBlbHNlIGlmIChwYXJhbXMkbW9kZWxpbmdfcGtnID09ICJ0aWR5bW9kZWxzIikgewogICAgICBtb2RlbF9yZXN1bHRzX2IgPC0gZml0VGlkeU1vZGVscyhYdHJhaW4gPSBYdHJhaW5fYiwgeXRyYWluID0geXRyYWluX2IsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBYdGVzdCA9IFh2YWxpZCwgeXRlc3QgPSB5dmFsaWQsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIG1vZGVsX2xpc3QgPSBtb2RlbF9saXN0LCBrZm9sZHMgPSBrZm9sZHMpCiAgICB9IGVsc2UgaWYgKHBhcmFtcyRtb2RlbGluZ19wa2cgPT0gImgybyIpIHsKICAgICAgbW9kZWxfcmVzdWx0c19iIDwtIGZpdGgybyhYdHJhaW4gPSBYdHJhaW5fYiwgeXRyYWluID0geXRyYWluX2IsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgWHRlc3QgPSBYdmFsaWQsIHl0ZXN0ID0geXZhbGlkLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIG1vZGVsX2xpc3QgPSBtb2RlbF9saXN0KQogICAgfQogICAgcmV0dXJuKG1vZGVsX3Jlc3VsdHNfYikKICB9LAogIHNpbXBsaWZ5ID0gRkFMU0UKKQoKYm9vdHN0cmFwX21vZGVsX2VycnMgPC0gcHVycnI6Om1hcF9kZnIoYm9vdHN0cmFwX21vZGVsX3Jlc3VsdHMsICJlcnJvcnMiLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLmlkID0gImJvb3RzdHJhcF9pZCIpCmJvb3RzdHJhcF9tb2RlbF9wcmVkcyA8LSBwdXJycjo6bWFwKGJvb3RzdHJhcF9tb2RlbF9yZXN1bHRzLCAicHJlZGljdGlvbnMiLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAuaWQgPSAiYm9vdHN0cmFwX2lkIikKYm9vdHN0cmFwX21vZGVsX2ltcHMgPC0gcHVycnI6Om1hcChib290c3RyYXBfbW9kZWxfcmVzdWx0cywgImltcG9ydGFuY2UiLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC5pZCA9ICJib290c3RyYXBfaWQiKSAlPiUKICBkcGx5cjo6YmluZF9yb3dzKC5pZCA9ICJib290c3RyYXBfaWQiKQoKYm9vdHN0cmFwX21vZGVsX2VycnNfc3VtbWFyeSA8LSBib290c3RyYXBfbW9kZWxfZXJycyAlPiUKICBkcGx5cjo6Z3JvdXBfYnkobW9kZWwpICU+JQogIGRwbHlyOjpzdW1tYXJpc2UoZHBseXI6OmFjcm9zcygtYm9vdHN0cmFwX2lkLCBsaXN0KG1lYW4gPSBtZWFuLCBzZCA9IHNkKSkpICU+JQogIHNldE5hbWVzKHN0cmluZ3I6OnN0cl9yZXBsYWNlKGNvbG5hbWVzKC4pLCAiXyIsICIgIikpCmBgYAoKYGBge3Igc3RhYmlsaXR5LWNoZWNrLXRhYmxlLCByZXN1bHRzID0gImFzaXMifQojIHRhYmxlIG9mIGFjY3VyYWN5IG1ldHJpY3MKcHJldHR5VGFibGUoCiAgYm9vdHN0cmFwX21vZGVsX2VycnNfc3VtbWFyeSwKICBkaWdpdHMgPSAyLCBzaWdmaWcgPSBGQUxTRSwgcm93bmFtZXMgPSBGQUxTRSwKICBjYXB0aW9uID0gIlZhbGlkYXRpb24gUHJlZGljdGlvbiBBY2N1cmFjaWVzIE92ZXIgQm9vdHN0cmFwcGVkIFRyYWluaW5nIEZpdHMiLAogIGJvbGRfZnVuY3Rpb24gPSAiLiA9PSBtYXgoLiwgbmEucm0gPSBUUlVFKSIsIGJvbGRfbWFyZ2luID0gMiwKICBib2xkX3NjaGVtZSA9IGMoRiwgVCwgRiwgVCwgRiksCiAgaHRtbF9vcHRpb25zID0gbGlzdChvcHRpb25zID0gbGlzdChkb20gPSAidCIpKQopCmBgYAoKYGBge3Igc3RhYmlsaXR5LWNoZWNrLWJveHBsb3R9CiMgYm94cGxvdHMKYm9vdHN0cmFwX21vZGVsX2VycnMgJT4lCiAgdGlkeXI6OnBpdm90X2xvbmdlcihjb2xzID0gLWMobW9kZWwsIGJvb3RzdHJhcF9pZCksIAogICAgICAgICAgICAgICAgICAgICAgbmFtZXNfdG8gPSAiTWV0cmljIiwgdmFsdWVzX3RvID0gIlZhbHVlIikgJT4lCiAgcGxvdEJveHBsb3QoeF9zdHIgPSAiVmFsdWUiLCB5X3N0ciA9ICJtb2RlbCIsIGhvcml6b250YWwgPSBGQUxTRSkgKwogIGdncGxvdDI6OmZhY2V0X3dyYXAofiBNZXRyaWMpICsKICBnZ3Bsb3QyOjpsYWJzKHggPSAiTWV0aG9kIiwgCiAgICAgICAgICAgICAgICB0aXRsZSA9ICJWYWxpZGF0aW9uIFByZWRpY3Rpb24gQWNjdXJhY2llcyBPdmVyIEJvb3RzdHJhcHBlZCBUcmFpbmluZyBGaXRzIikKYGBgCgojIyBJbnRlcnByZXRhYmlsaXR5IHsudGFic2V0IC5jdXN0b20tdGFic30KCkZvciB0aGUgbW9kZWxzIHRoYXQgcGFzcyB0aGUgcHJlZGljdGlvbiBhbmQgc3RhYmlsaXR5IGNoZWNrcywgZXh0cmFjdCB0aGUgaW1wb3J0YW50IGZlYXR1cmVzIGluIHRoZSBwcmVkaWN0aXZlIG1vZGVscyB0aGF0IGFyZSBzdGFibGUgYWNyb3NzIGJvdGggZGF0YSBhbmQgbW9kZWwgcGVydHVyYmF0aW9ucy4gRGV0ZXJtaW5pbmcgdGhlIGltcG9ydGFuY2Ugb2YgYSBmZWF0dXJlIGNhbiBiZSBtZXRob2QgZGVwZW5kZW50LgoKYGBge2FzaXMsIGludGVyYWN0aXZlX3RleHQgPSBUUlVFfQoKYGBgCgojIyMgRnVsbCBNb2RlbCAod2l0aG91dCBzdGFiaWxpdHkpIHsudW5udW1iZXJlZCAudGFic2V0IC50YWJzZXQtcGlsbHN9CgojIyMjIFRhYmxlIHsudW5udW1iZXJlZH0KCmBgYHtyIGludGVycHJldGFiaWxpdHktdGFibGUsIGFkZF9uZXdfbGluZSA9IFRSVUUsIHJlc3VsdHMgPSAiYXNpcyJ9CnByZXR0eVRhYmxlKG1vZGVsX3Jlc3VsdHMkaW1wb3J0YW5jZSwgZGlnaXRzID0gMiwgc2lnZmlnID0gRkFMU0UsCiAgICAgICAgICAgIGNhcHRpb24gPSAiVmFyaWFibGUgSW1wb3J0YW5jZXMiKQpgYGAKCiMjIyMgUGxvdHMgey51bm51bWJlcmVkfQoKYGBge3IgaW50ZXJwcmV0YWJpbGl0eS1wbG90LCBhZGRfbmV3X2xpbmUgPSBUUlVFfQpwbG90RmVhdHVyZUltcG9ydGFuY2UobW9kZWxfcmVzdWx0cyRpbXBvcnRhbmNlLAogICAgICAgICAgICAgICAgICAgICAgdXNlX3JhbmtpbmdzID0gRkFMU0UsCiAgICAgICAgICAgICAgICAgICAgICB1c2VfZmFjZXRzID0gVFJVRSwKICAgICAgICAgICAgICAgICAgICAgIGludGVyYWN0aXZlID0gRkFMU0UpCmBgYAoKYGBge3IgaW50ZXJwcmV0YWJpbGl0eS1wYWlyLXBsb3QsIGFkZF9uZXdfbGluZSA9IFRSVUV9CnBsb3RGZWF0dXJlSW1wb3J0YW5jZVBhaXIobW9kZWxfcmVzdWx0cyRpbXBvcnRhbmNlLAogICAgICAgICAgICAgICAgICAgICAgICAgIHVzZV9yYW5raW5ncyA9IFRSVUUsCiAgICAgICAgICAgICAgICAgICAgICAgICAgaW50ZXJhY3RpdmUgPSBGQUxTRSkKYGBgCgojIyMgQm9vdHN0cmFwcGVkIE1vZGVsICh3aXRoIHN0YWJpbGl0eSkgey51bm51bWJlcmVkIC50YWJzZXQgLnRhYnNldC1waWxsc30KCiMjIyMgVGFibGUgey51bm51bWJlcmVkfQoKYGBge3IgYm9vdC1pbnRlcnByZXRhYmlsaXR5LXRhYmxlLCBhZGRfbmV3X2xpbmUgPSBUUlVFLCByZXN1bHRzID0gImFzaXMifQpib290c3RyYXBfbW9kZWxfaW1wc19zdW1tYXJ5IDwtIGJvb3RzdHJhcF9tb2RlbF9pbXBzICU+JQogIGRwbHlyOjpncm91cF9ieShtb2RlbCwgdmFyaWFibGUpICU+JQogIGRwbHlyOjpzdW1tYXJpc2UoYE1lYW4gSW1wb3J0YW5jZWAgPSBtZWFuKGltcG9ydGFuY2UpLAogICAgICAgICAgICAgICAgICAgYE1lZGlhbiBJbXBvcnRhbmNlYCA9IG1lZGlhbihpbXBvcnRhbmNlKSwKICAgICAgICAgICAgICAgICAgIGBTRCBJbXBvcnRhbmNlYCA9IHNkKGltcG9ydGFuY2UpLAogICAgICAgICAgICAgICAgICAgYE1pbiBJbXBvcnRhbmNlYCA9IG1pbihpbXBvcnRhbmNlKSwKICAgICAgICAgICAgICAgICAgIGBNYXggSW1wb3J0YW5jZWAgPSBtYXgoaW1wb3J0YW5jZSkpCnByZXR0eVRhYmxlKAogIGJvb3RzdHJhcF9tb2RlbF9pbXBzX3N1bW1hcnksIAogIGRpZ2l0cyA9IDIsIHNpZ2ZpZyA9IEYsIAogIGNhcHRpb24gPSAiU3VtbWFyeSBvZiB2YXJpYWJsZSBpbXBvcnRhbmNlcyBhY3Jvc3MgYm9vdHN0cmFwcGVkIG1vZGVscyIKKQpgYGAKCiMjIyMgUGxvdHMgey51bm51bWJlcmVkfQoKYGBge3IgYm9vdC1pbnRlcnByZXRhYmlsaXR5LXBsb3QsIGFkZF9uZXdfbGluZSA9IFRSVUV9CnBsb3RGZWF0dXJlSW1wb3J0YW5jZVN0YWJpbGl0eShib290c3RyYXBfbW9kZWxfaW1wcywKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHVzZV9yYW5raW5ncyA9IEZBTFNFLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgdXNlX2ZhY2V0cyA9IEZBTFNFLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgaW50ZXJhY3RpdmUgPSBGQUxTRSkKYGBgCgojIE1haW4gUmVzdWx0cwoKSW50ZXJwcmV0IGFuZCBzdW1tYXJpemUgdGhlIHByZWRpY3Rpb24gYW5kIHN0YWJpbGl0eSByZXN1bHRzLgoKYGBge2FzaXMsIGludGVyYWN0aXZlX3RleHQgPSBUUlVFfQoKYGBgCgpFdmFsdWF0ZSBwaXBlbGluZSBvbiB0ZXN0IGRhdGEuCgpgYGB7ciBmaW5hbC1maXRzLCBldmFsID0gVFJVRSB9Clh0cmFpbl9maW5hbCA8LSBkcGx5cjo6YmluZF9yb3dzKFh0cmFpbiwgWHZhbGlkKQp5dHJhaW5fZmluYWwgPC0gYyh5dHJhaW4sIHl2YWxpZCkKaWYgKHBhcmFtcyRtb2RlbGluZ19wa2cgPT0gImNhcmV0IikgewogIGZpbmFsX21vZGVsX3Jlc3VsdHMgPC0gZml0Q2FyZXQoWHRyYWluID0gWHRyYWluX2ZpbmFsLCB5dHJhaW4gPSB5dHJhaW5fZmluYWwsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgWHRlc3QgPSBYdGVzdCwgeXRlc3QgPSB5dGVzdCwgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBtb2RlbF9saXN0ID0gbW9kZWxfbGlzdCwgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB0cl9jb250cm9sID0gdHJfY29udHJvbCkKfSBlbHNlIGlmIChwYXJhbXMkbW9kZWxpbmdfcGtnID09ICJ0aWR5bW9kZWxzIikgewogIGZpbmFsX21vZGVsX3Jlc3VsdHMgPC0gZml0VGlkeU1vZGVscyhYdHJhaW4gPSBYdHJhaW5fZmluYWwsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB5dHJhaW4gPSB5dHJhaW5fZmluYWwsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBYdGVzdCA9IFh0ZXN0LCB5dGVzdCA9IHl0ZXN0LAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBtb2RlbF9saXN0ID0gbW9kZWxfbGlzdCwga2ZvbGRzID0ga2ZvbGRzKQp9IGVsc2UgaWYgKHBhcmFtcyRtb2RlbGluZ19wa2cgPT0gImgybyIpIHsKICBmaW5hbF9tb2RlbF9yZXN1bHRzIDwtIGZpdGgybyhYdHJhaW4gPSBYdHJhaW5fZmluYWwsIHl0cmFpbiA9IHl0cmFpbl9maW5hbCwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBYdGVzdCA9IFh0ZXN0LCB5dGVzdCA9IHl0ZXN0LAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIG1vZGVsX2xpc3QgPSBtb2RlbF9saXN0KQp9CgpgYGAKClN1bW1hcml6ZSB0ZXN0IHNldCBwcmVkaWN0aW9uIGFuZC9vciBpbnRlcnByZXRhYmlsaXR5IHJlc3VsdHMuCgpgYGB7YXNpcywgaW50ZXJhY3RpdmVfdGV4dCA9IFRSVUV9CgpgYGAKCjwhLS0gVE9ETzogQW5hIC0gYWRkIHRlbXBsYXRlIHRhYmxlcyB3aXRoIGludGVycHJldGF0aW9uIC0tPgoKYGBge3IgdGVzdC1hY2N1cmFjeS10YWJsZSwgcmVzdWx0cyA9ICJhc2lzIn0KIyB0YWJsZSBvZiBhY2N1cmFjeSBtZXRyaWNzCnByZXR0eVRhYmxlKGZpbmFsX21vZGVsX3Jlc3VsdHMkZXJyb3JzLCAKICAgICAgICAgICAgZGlnaXRzID0gMiwgc2lnZmlnID0gRkFMU0UsIHJvd25hbWVzID0gRkFMU0UsCiAgICAgICAgICAgIGNhcHRpb24gPSAiVGVzdCBQcmVkaWN0aW9uIEFjY3VyYWNpZXMiLCAKICAgICAgICAgICAgaHRtbF9vcHRpb25zID0gbGlzdChvcHRpb25zID0gbGlzdChkb20gPSAidCIpKSkKYGBgCgojIFBvc3QgaG9jIGFuYWx5c2lzCgpNb3ZlIGJleW9uZCB0aGUgZ2xvYmFsIHByZWRpY3Rpb24gYWNjdXJhY3kgbWV0cmljcyBhbmQgZGl2ZSBkZWVwZXIgaW50byBpbmRpdmlkdWFsLWxldmVsIHByZWRpY3Rpb25zIGZvciB0aGUgdmFsaWRhdGlvbiBhbmQvb3IgdGVzdCBzZXQsIGkuZS4sIHByb3ZpZGUgYSBtb3JlICJsb2NhbCIgYW5hbHlzaXMuCgotICAgRXhhbWluZSBhbnkgcG9pbnRzIHRoYXQgaGFkIHBvb3IgcHJlZGljdGlvbnMuCi0gICBFeGFtaW5lIGRpZmZlcmVuY2VzIGJldHdlZW4gcHJlZGljdGlvbiBtZXRob2RzLgoKYGBge2FzaXMsIGludGVyYWN0aXZlX3RleHQgPSBUUlVFfQoKYGBgCgpgYGB7ciBwb3N0aG9jLXBhaXItcGxvdH0KcGxvdFBhaXJzKGNiaW5kKGBUcnVlIFJlcHNvbnNlc2AgPSB5dGVzdCwgZmluYWxfbW9kZWxfcmVzdWx0cyRwcmVkaWN0aW9ucyksIAogICAgICAgICAgY29sdW1ucyA9IDE6KG5jb2woZmluYWxfbW9kZWxfcmVzdWx0cyRwcmVkaWN0aW9ucykgKyAxKSwgCiAgICAgICAgICB0aXRsZSA9ICJDb21wYXJpc29uIG9mIG1vZGVsIHByZWRpY3Rpb25zIikKYGBgCgojIENvbmNsdXNpb25zCgpSZWl0ZXJhdGUgbWFpbiBmaW5kaW5ncywgbm90ZSBhbnkgY2F2ZWF0cywgYW5kIGNsZWFybHkgdHJhbnNsYXRlIGZpbmRpbmdzL2FuYWx5c2lzIGJhY2sgdG8gdGhlIGRvbWFpbiBwcm9ibGVtIGNvbnRleHQuCgpgYGB7YXNpcywgaW50ZXJhY3RpdmVfdGV4dCA9IFRSVUV9CgpgYGAKCg==

PCS Documentation

December 22, 2021

1 Domain problem formulation

2 Data

2.1 Data Collection

2.2 Data Splitting

2.2.1 Data Splitting Overview

X Data Split

Y Data Split

2.3 Data Cleaning and Preprocessing

2.4 Data Exploration

2.4.1 Data Overview

Summary Tables

X Distribution

Y Distribution

Data Heatmap

Feature Correlation

Feature Pair Plots

Marginal Associations

PCA

3 Prediction Modeling

3.1 Prediction check

3.2 Stability check

3.3 Interpretability

Full Model (without stability)

Table

Plots

Bootstrapped Model (with stability)

Table

Plots

4 Main Results

5 Post hoc analysis

6 Conclusions